Skip to content

test(expo): comprehensive test coverage for native components#8334

Open
chriscanin wants to merge 66 commits into
mainfrom
chris/native-component-tests
Open

test(expo): comprehensive test coverage for native components#8334
chriscanin wants to merge 66 commits into
mainfrom
chris/native-component-tests

Conversation

@chriscanin
Copy link
Copy Markdown
Member

Description

Adds comprehensive test coverage for @clerk/expo native components across three layers, each targeting a specific class of regression.

Backstory: the recent SSO/profile/theming work (chris/fix-inline-authview-sso) shipped four user-visible bugs and fixes (iOS forgot-password OAuth, Android Get Help loop, cold-launch white flash, native theming reset). Zero automated tests existed to catch any of them. This PR establishes the infrastructure.

What's in the PR

JS unit tests (packages/expo/src/**/__tests__/) — 20 new files, 216 tests. Full coverage of every previously untested module: hooks (useUserProfileModal, useNativeAuthEvents, useNativeSession), native component wrappers (AuthView, InlineAuthView, UserButton, UserProfileView, InlineUserProfileView), provider (ClerkProvider init flow, NativeSessionSync, native-to-JS auth sync), utilities, caches, and the Expo config plugin.

Android (Kotlin) unit tests (packages/expo/android/src/test/) — 3 files, 8 tests. Covers session-ID change detection logic, per-view ViewModelStore isolation, and sign-out cleanup behavior. Targets the logic fixed in the Android regression commits.

iOS (Swift) unit tests (packages/expo/ios/Tests/) — 2 files, 13 tests. Covers the viewDidDisappear session-ID comparison (the cancel-vs-success decision), the presentWhenReady guard predicate (attempts cap + invalidation), and the emitAuthStateChange payload shape.

Maestro e2e flows (integration-mobile/flows/) — 23 YAML files targeting the clerk-expo-quickstart NativeComponentQuickstart app. Includes 5 regression flows:

  • flows/sign-in/google-sso-from-forgot-password.yaml — iOS OAuth from forgot-password
  • flows/sign-in/get-help-loop-regression.yaml — Android AuthView navigation loop
  • flows/cycles/sign-in-sign-out-sign-in.yaml — inline AuthView re-sign-in
  • flows/theming/custom-theme-applied.yaml — native theming reset
  • flows/smoke/cold-launch-no-flash.yaml — cold-launch white flash

Plus 11 happy-path flows and 6 reusable subflows.

CI workflow (.github/workflows/mobile-e2e.yml) — manual workflow_dispatch trigger. Clones clerk-expo-quickstart at a configurable ref, builds on macos-15 (iOS) and ubuntu-latest with reactivecircus/android-emulator-runner (Android), runs all non-manual Maestro flows. Required secrets: CLERK_TEST_PK, CLERK_TEST_EMAIL, CLERK_TEST_PASSWORD.

Source changes (non-breaking)

  • packages/expo/app.plugin.js: named exports for withClerkIOS, withClerkAndroid, withClerkAppleSignIn, withClerkGoogleSignIn, withClerkKeychainService (additive, default export unchanged)
  • packages/expo/src/provider/ClerkProvider.tsx: NativeSessionSync marked as exported for test access (internal, documented as not public API)
  • packages/expo/android/build.gradle: JUnit/Robolectric/MockK test dependencies + testOptions for Robolectric
  • packages/expo/ios/ClerkExpo.podspec: test_spec 'Tests' block so Cocoapods generates the test target

How to test

JS unit tests run in existing CI:

cd packages/expo && pnpm test
# 24 files, 216 tests passing

Native unit tests:

# Android
cd packages/expo/android && ./gradlew :clerk_expo:test

# iOS (after pod install in a consuming app)
xcodebuild test -workspace <path>/ios/Pods/Pods.xcworkspace -scheme ClerkExpo-Unit-Tests

Maestro flows:

# Local (requires clerk-expo-quickstart cloned as sibling + Maestro CLI installed)
cd integration-mobile
cp config/.env.example config/.env  # fill in values
./scripts/run-android.sh   # or run-ios.sh, or run-all.sh

CI: trigger the Mobile e2e (@clerk/expo) workflow manually from the Actions tab.

Checklist

  • pnpm test runs as expected (216 tests passing).
  • pnpm build runs as expected.
  • (If applicable) JSDoc comments have been added or updated for any package exports
  • (If applicable) Documentation has been updated

Type of change

  • 📖 Refactoring / dependency upgrade / documentation (testing infrastructure only, no runtime behavior changes)

Add 216 JS unit tests across 20 new test files covering every untested
module in @clerk/expo: hooks (useUserProfileModal, useNativeAuthEvents,
useNativeSession), native components (AuthView, InlineAuthView,
UserProfileView, InlineUserProfileView, UserButton), provider
(ClerkProvider init flow, NativeSessionSync, native-to-JS auth sync),
utilities (runtime, errors, native-module), caches (token-cache,
resource-cache), and the Expo config plugin (withClerkAndroid,
withClerkExpo, withClerkIOS).

Add 8 Kotlin unit tests for the Android native bridge code covering
session ID change detection logic, per-view ViewModelStore isolation,
and sign-out cleanup behavior.

Add 23 Maestro e2e flow files targeting the clerk-expo-quickstart
NativeComponentQuickstart app, including 5 regression flows for bugs
shipped in chris/fix-inline-authview-sso (forgot-password OAuth,
Get Help loop, re-sign-in cycle, theming reset, cold-launch flash).

Add manual-trigger GitHub Actions workflow for running Maestro flows
on both iOS simulator and Android emulator.

Source changes (non-breaking):
- packages/expo/app.plugin.js: export sub-plugins for unit testing
- packages/expo/src/provider/ClerkProvider.tsx: export NativeSessionSync
- packages/expo/android/build.gradle: add JUnit/Robolectric test deps
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 16, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
clerk-js-sandbox Ready Ready Preview, Comment May 22, 2026 7:17pm

Request Review

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 16, 2026

🦋 Changeset detected

Latest commit: 1aab538

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@clerk/expo Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Comment thread .github/workflows/mobile-e2e.yml Fixed
Comment thread .github/workflows/mobile-e2e.yml Fixed
Comment on lines +138 to +145
run: |
cd clerk-expo-quickstart/NativeComponentQuickstart
npx expo prebuild --clean
npx expo run:ios --configuration Release --no-bundler
cd ../../integration-mobile
source config/.env 2>/dev/null || true
maestro test --exclude-tags "${{ inputs.exclude_tags }},androidOnly" flows/

Copy link
Copy Markdown

@semgrep-code-clerk semgrep-code-clerk Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".

Fixed in commit fe9e3fe

chriscanin added a commit to clerk/clerk-android that referenced this pull request Apr 16, 2026
Introduces an `expo-compat` job in the manual-release workflow that
runs before `publish`. The job:

1. Publishes the current SDK source to mavenLocal with a snapshot suffix
2. Clones clerk/javascript and clerk/clerk-expo-quickstart
3. Patches @clerk/expo's pinned clerk-android version to the snapshot
4. Adds mavenLocal() to the gradle repositories so resolution works
5. Builds the quickstart NativeComponentQuickstart against the snapshot
6. Runs the Maestro e2e suite from clerk/javascript's integration-mobile/

The `publish` job now depends on `expo-compat` succeeding, so a
release cannot publish if the Expo integration tests fail.

Secrets required (to be configured on this repo):
- CLERK_TEST_EMAIL
- CLERK_TEST_PASSWORD
- EXPO_PUBLIC_CLERK_PUBLISHABLE_KEY

Related: clerk/javascript#8334 (adds the
integration-mobile/ test suite this workflow invokes)
chriscanin added a commit to clerk/clerk-ios that referenced this pull request Apr 16, 2026
Introduces an `expo-compat` job in release-sdk.yml that runs between
`checks` and `publish`. The job validates that the clerk-ios SHA about
to be published does not break @clerk/expo's native component integration.

The job:

1. Clones clerk/javascript and clerk/clerk-expo-quickstart
2. Patches packages/expo/app.plugin.js to pin the SPM clerk-ios dependency
   to the current release SHA using requirement kind 'revision' instead
   of 'exactVersion'
3. Builds the NativeComponentQuickstart app via `expo run:ios --configuration Release`
4. Runs the Maestro e2e suite from integration-mobile/ on an iOS simulator
5. If any Maestro flow fails, the `publish` job is blocked

Because the clerk-ios dependency is resolved via SPM, no local publish
step is needed — SPM clones the clerk-ios repo at the specified SHA
during the quickstart's Xcode build.

Secrets required:
- CLERK_TEST_EMAIL
- CLERK_TEST_PASSWORD
- EXPO_PUBLIC_CLERK_PUBLISHABLE_KEY

Related:
- clerk/javascript#8334 — adds the integration-mobile/ test suite
- clerk/clerk-android#593 — Android equivalent of this gate
Local iOS validation surfaced several issues in the Maestro flow files
and runner scripts. This commit has all the fixes needed to get the
core happy-path and regression flows passing end-to-end against the
clerk-expo-quickstart NativeComponentQuickstart app on an iPhone 17
simulator (iOS 26).

Validated passing flows:
- flows/sign-in/email-password.yaml (34s)
- flows/cycles/sign-in-sign-out-sign-in.yaml (53s) -- THE REGRESSION
- flows/smoke/cold-launch-no-flash.yaml (7s)

Remaining flows need follow-up iteration to handle iOS-specific
UserProfile UI copy (e.g. Edit profile, Log out button text) and
the secondary test user env vars for different-user cycles.

Fixes in this commit:

1. Scripts portability -- macOS ships bash 3.2 which lacks mapfile.
   Replace with while-read loop.

2. Maestro subdirectory recursion -- `maestro test flows/` does not
   walk subdirectories. Use `find` + explicit file list.

3. Platform disambiguation -- with both iOS sim and Android emu booted,
   Maestro auto-picked the wrong driver. Pass `--platform ios|android`.

4. Env var interpolation -- Maestro does not auto-read shell env. Pass
   CLERK_TEST_EMAIL/PASSWORD via explicit `-e KEY=value` flags.

5. Regex patterns -- Maestro's `text:` and `visible:` use full-string
   regex match. Use `.*term.*` for substring, `\.?` for optional
   trailing punctuation, single quotes in YAML to avoid escape issues.

6. Dev launcher URL differs -- iOS uses http://localhost:8081, Android
   uses http://10.0.2.2:8081. Match with `.*:8081` regex.

7. Dev menu dismissal -- tap Close accessibility ID with backdrop
   fallback at 50%,20%.

8. Session persistence across clearState -- Clerk's token in iOS
   Keychain (AFTER_FIRST_UNLOCK) survives app reinstall. Add a
   conditional sign-out step to open-app.yaml.

9. inputText appends, not replaces -- add `eraseText: 50` before every
   inputText in sign-in-email-password.yaml.

10. iOS trailing period differs -- clerk-ios renders "Welcome! Sign in
    to continue" (no period), clerk-android renders with period. Use
    `\.?` regex to match both.

Also adds integration-mobile/.gitignore to prevent config/.env from
being committed (it contains a Clerk publishable key for the
delicate-crab-73 dev instance).
Comment on lines +140 to +149
run: |
cd clerk-expo-quickstart/NativeComponentQuickstart
npx expo prebuild --clean
npx expo run:ios --configuration Release --no-bundler
cd ../../integration-mobile
source config/.env 2>/dev/null || true
# Maestro doesn't auto-recurse into subdirectories; pass each flow explicitly.
find flows -type f -name "*.yaml" ! -path "*/common/*" -print0 | \
xargs -0 maestro test --exclude-tags "${{ inputs.exclude_tags }},androidOnly"

Copy link
Copy Markdown

@semgrep-code-clerk semgrep-code-clerk Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".

🧹 Fixed in commit 4b7c966 🧹

iOS UserProfile uses different copy than Android:
- "Edit profile" (Android) -> "Update profile" (iOS)
- "Log out" (Android) -> "Sign out" (iOS)
- The Close (X) button matches on accessibilityText "Close", not id

Use cross-platform regex alternation ("(Edit|Update) profile",
"Log out|Sign out") and switch from `id: "Close"` to `text: "Close"`
since Maestro's id matches resource-id (SF Symbol name "xmark" on iOS).

Also switch sheet-dismiss from `- back` (iOS has no back button) to
tap the Close X with back fallback for Android.

Mark 3 flows as `skip` until prerequisites are in place:
- sign-out-then-sign-in-different-user: needs CLERK_TEST_EMAIL_SECONDARY
  and a second test user in the dev instance
- email-verification: sign-up selector flow still needs iOS-specific
  verification steps
- custom-theme-applied: check-theme-color.js needs pngjs, and iOS
  quickstart doesn't bundle clerk-theme.json yet

Passing flows on iPhone 17 simulator:
- email-password
- sign-in-sign-out-sign-in (THE REGRESSION)
- cold-launch-no-flash
- open-profile-modal
- sign-out-from-profile
- edit-first-name
cold-launch-no-flash inlines its own launcher logic (doesn't use
open-app.yaml) so it was missing the conditional sign-out step added
to open-app.yaml. When the previous flow left the user signed in, the
cold-launch assertion "Welcome! Sign in to continue" failed because
the app launched to the signed-in home screen.

Also update the dev menu dismissal to use the same Close-X-first,
backdrop-fallback pattern as open-app.yaml.

Result: 6/6 non-skipped iOS Maestro flows passing in 4m 14s on
iPhone 17 simulator (iOS 26) against delicate-crab-73 dev instance:
- email-password
- sign-in-sign-out-sign-in (the shipped regression)
- cold-launch-no-flash
- open-profile-modal
- sign-out-from-profile
- edit-first-name
Add Google Password Manager auto-dismissal to open-app.yaml and
sign-in-email-password.yaml. After sign-in, Android shows a "Save
password?" sheet from Google Password Manager. The sheet button text
varies between "Not now" (first prompt) and "Never" (after declining
once), so use regex alternation.

Skip dark-mode-applied -- same pngjs dependency issue as
custom-theme-applied; both need the theme-color helper script
prerequisites before they can run.

Result: 7/7 non-skipped Android Maestro flows passing against
Pixel 9 Pro emulator (API 34) and delicate-crab-73 dev instance:
- email-password (57s)
- sign-in-sign-out-sign-in (1m 28s) -- the shipped regression
- cold-launch-no-flash (24s)
- get-help-loop-regression (1m 10s) -- the shipped Android regression
- open-profile-modal (1m 9s)
- sign-out-from-profile (1m 4s)
- edit-first-name (1m 16s)

Combined with iOS (6/6 passing), the Maestro suite now catches the
full user journey end-to-end on both platforms.
Mirrors the /integration (Playwright) secret pattern: read pk/sk from a
named entry in the existing INTEGRATION_INSTANCE_KEYS JSON secret and
provision a fresh test user per run via the Clerk Backend API. Cleans up
the user on teardown (always).

Instance name is a placeholder ("expo-native") pending SDK team confirmation
of which dev/staging instance this workflow should target. The secret slot
is left blank in the repo until that's resolved.
Comment thread .github/workflows/mobile-e2e.yml Fixed
Comment on lines +244 to +252
run: |
cd clerk-expo-quickstart/NativeComponentQuickstart
npx expo prebuild --clean
npx expo run:ios --configuration Release --no-bundler
cd ../../integration-mobile
# Maestro doesn't auto-recurse into subdirectories; pass each flow explicitly.
find flows -type f -name "*.yaml" ! -path "*/common/*" -print0 | \
xargs -0 maestro test --exclude-tags "${{ inputs.exclude_tags }},androidOnly"

Copy link
Copy Markdown

@semgrep-code-clerk semgrep-code-clerk Bot Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using variable interpolation ${{...}} with github context data in a run: step could allow an attacker to inject their own code into the runner. This would allow them to steal secrets and code. github context data can have arbitrary user input and should be treated as untrusted. Instead, use an intermediate environment variable with env: to store the data and use the environment variable in the run: script. Be sure to use double-quotes the environment variable, like this: "$ENVVAR".

🎉 Removed in commit 808601a 🎉

@chriscanin chriscanin marked this pull request as ready for review May 6, 2026 21:00
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 6, 2026

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR introduces a comprehensive test infrastructure and mobile e2e automation framework for the Expo package. The changeset documents the export of NativeSessionSync and test utilities. Android and iOS build configurations add unit test dependencies (JUnit, Robolectric, MockK, XCTest). The app.plugin.js exports named plugin functions and testing utilities. Native unit tests cover session detection logic (Android) and payload/presentation/comparison logic (iOS). JavaScript tests verify hooks (useNativeAuthEvents, useNativeSession, useUserProfileModal), ClerkProvider native initialization and sync flows, and utility functions (resource cache, token cache, error validation). Native view components (AuthView, InlineAuthView, UserButton, UserProfileView, InlineUserProfileView) are tested for rendering, prop forwarding, event handling, and sign-out flows. Plugin tests verify configuration queueing and idempotency. The mobile e2e workflow is substantially enhanced with compat-gate pinning for clerk-ios/android versions, binary source hash caching, improved Maestro test user provisioning via staging BAPI, and comprehensive flow definitions covering sign-in, sign-up, profile, regression, smoke, and theming scenarios. Test orchestration scripts enable platform-scoped and filtered flow execution. Supporting changes improve documentation links and error messaging.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

mac runners default /bin/bash to Apple's 3.2 fork, which lacks
mapfile. Use the same find | while | grep | xargs pipeline as the
Android job; skip-messages go to stderr so xargs only sees kept paths.
…arallel

Previous concurrency group was keyed only by github.ref, so dispatches
from both iOS and Android compat gates against the same receiver branch
share the same group. With cancel-in-progress: true, firing the Android
gate cancels an in-flight iOS run (and vice-versa).

Scope the group by which platform's compat gate fired the dispatch
(ios / android / full) so the two can run concurrently. Rapid
re-dispatch within the same scope still cancels (intended behavior).
The folded scalar (>-) only joins same-indent lines with spaces;
more-indented lines preserve newlines, which broke the shell's
while/do/done pairing and produced 'end of file unexpected expecting done'.
@chriscanin chriscanin marked this pull request as ready for review May 20, 2026 21:58
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/workflows/mobile-e2e.yml (1)

101-103: ⚖️ Poor tradeoff

Consider pinning actions to SHA hashes for supply chain security.

Static analysis flagged multiple unpinned action references throughout this workflow (e.g., actions/checkout@v4, actions/setup-java@v4, actions/cache@v4). Pinning to specific commit SHAs prevents potential supply chain attacks if an action's release tag is compromised.

This may be a repo-wide policy decision. If pinning is desired, tools like pin-github-action or Dependabot's grouped updates can help maintain pinned versions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/mobile-e2e.yml around lines 101 - 103, The workflow uses
third-party actions with floating tags (e.g., actions/checkout@v4,
actions/setup-java@v4, actions/cache@v4); replace each tag reference with the
corresponding commit SHA to pin the action (obtain the SHA from the action
repo's tag or release page) so the workflow references e.g.,
actions/checkout@<commit-sha> instead of `@v4`, and apply the same change for
actions/setup-java and actions/cache to harden supply-chain security; optionally
document the SHAs in a comment and use a tool like pin-github-action or
Dependabot grouped updates to keep pinned SHAs up to date.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/mobile-e2e.yml:
- Around line 639-646: The step "Compute binary source hash" (id: bin-hash) uses
sha256sum which is not available on macOS runners; replace the sha256sum
invocation with the portable shasum -a 256 and extract the first 16 hex chars
(e.g. pipe to awk '{print substr($1,1,16)}' or use cut -c1-16) so the computed
value assigned to the hash variable remains the same and works on both Linux and
macOS.

---

Nitpick comments:
In @.github/workflows/mobile-e2e.yml:
- Around line 101-103: The workflow uses third-party actions with floating tags
(e.g., actions/checkout@v4, actions/setup-java@v4, actions/cache@v4); replace
each tag reference with the corresponding commit SHA to pin the action (obtain
the SHA from the action repo's tag or release page) so the workflow references
e.g., actions/checkout@<commit-sha> instead of `@v4`, and apply the same change
for actions/setup-java and actions/cache to harden supply-chain security;
optionally document the SHAs in a comment and use a tool like pin-github-action
or Dependabot grouped updates to keep pinned SHAs up to date.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 6c02f857-94ff-4d6d-96df-f2c9f63ebf39

📥 Commits

Reviewing files that changed from the base of the PR and between e4ec3ef and 82b3e0f.

📒 Files selected for processing (34)
  • .github/workflows/mobile-e2e.yml
  • integration/mobile/.gitignore
  • integration/mobile/config/.env.example
  • integration/mobile/fixtures/test-users.json
  • integration/mobile/flows/common/assert-signed-in.yaml
  • integration/mobile/flows/common/assert-signed-out.yaml
  • integration/mobile/flows/common/open-app.yaml
  • integration/mobile/flows/common/sign-in-email-password.yaml
  • integration/mobile/flows/common/sign-out-via-button.yaml
  • integration/mobile/flows/common/sign-out-via-profile.yaml
  • integration/mobile/flows/cycles/sign-in-sign-out-sign-in.yaml
  • integration/mobile/flows/cycles/sign-out-then-sign-in-different-user.yaml
  • integration/mobile/flows/profile/edit-first-name.yaml
  • integration/mobile/flows/profile/open-inline-profile.yaml
  • integration/mobile/flows/profile/open-profile-modal.yaml
  • integration/mobile/flows/profile/sign-out-from-profile.yaml
  • integration/mobile/flows/sign-in/apple.yaml
  • integration/mobile/flows/sign-in/email-password.yaml
  • integration/mobile/flows/sign-in/get-help-loop-regression.yaml
  • integration/mobile/flows/sign-in/github.yaml
  • integration/mobile/flows/sign-in/google-sso-from-forgot-password.yaml
  • integration/mobile/flows/sign-in/google-sso-from-main.yaml
  • integration/mobile/flows/sign-up/email-verification.yaml
  • integration/mobile/flows/sign-up/google-sso-new-user.yaml
  • integration/mobile/flows/smoke/cold-launch-no-flash.yaml
  • integration/mobile/flows/theming/custom-theme-applied.yaml
  • integration/mobile/flows/theming/dark-mode-applied.yaml
  • integration/mobile/scripts/bootstrap-test-app.sh
  • integration/mobile/scripts/check-theme-color.js
  • integration/mobile/scripts/install-maestro.sh
  • integration/mobile/scripts/run-all.sh
  • integration/mobile/scripts/run-android.sh
  • integration/mobile/scripts/run-ios.sh
  • integration/mobile/scripts/run-regressions.sh
💤 Files with no reviewable changes (28)
  • integration/mobile/.gitignore
  • integration/mobile/config/.env.example
  • integration/mobile/flows/profile/open-inline-profile.yaml
  • integration/mobile/scripts/install-maestro.sh
  • integration/mobile/flows/common/assert-signed-in.yaml
  • integration/mobile/flows/sign-in/github.yaml
  • integration/mobile/flows/profile/open-profile-modal.yaml
  • integration/mobile/flows/sign-in/google-sso-from-main.yaml
  • integration/mobile/scripts/check-theme-color.js
  • integration/mobile/scripts/run-all.sh
  • integration/mobile/flows/cycles/sign-out-then-sign-in-different-user.yaml
  • integration/mobile/flows/sign-in/email-password.yaml
  • integration/mobile/flows/sign-up/google-sso-new-user.yaml
  • integration/mobile/fixtures/test-users.json
  • integration/mobile/flows/sign-in/apple.yaml
  • integration/mobile/flows/sign-in/google-sso-from-forgot-password.yaml
  • integration/mobile/flows/sign-up/email-verification.yaml
  • integration/mobile/scripts/run-ios.sh
  • integration/mobile/scripts/run-regressions.sh
  • integration/mobile/flows/common/sign-out-via-profile.yaml
  • integration/mobile/scripts/bootstrap-test-app.sh
  • integration/mobile/flows/common/assert-signed-out.yaml
  • integration/mobile/flows/theming/dark-mode-applied.yaml
  • integration/mobile/flows/smoke/cold-launch-no-flash.yaml
  • integration/mobile/flows/theming/custom-theme-applied.yaml
  • integration/mobile/scripts/run-android.sh
  • integration/mobile/flows/profile/edit-first-name.yaml
  • integration/mobile/flows/sign-in/get-help-loop-regression.yaml
✅ Files skipped from review due to trivial changes (1)
  • integration/mobile/flows/common/sign-out-via-button.yaml

Comment thread .github/workflows/mobile-e2e.yml Outdated
Comment on lines +639 to +646
- name: Compute binary source hash
id: bin-hash
run: |
expo_tree=$(git ls-tree -r HEAD packages/expo .github/workflows/mobile-e2e.yml | grep -v "/dist/")
qs_tree=$(git -C clerk-expo-quickstart ls-tree -r HEAD NativeComponentQuickstart | grep -vE "node_modules|/android/|/ios/")
hash=$(printf '%s\n%s\n' "$expo_tree" "$qs_tree" | sha256sum | cut -c1-16)
echo "hash=$hash" >> "$GITHUB_OUTPUT"
echo "Binary source hash: $hash"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify if sha256sum is used elsewhere in the repo's workflows and whether
# there's an established pattern for cross-platform hashing.

echo "=== Checking sha256sum usage in workflows ==="
rg -n 'sha256sum' .github/workflows/ 2>/dev/null || echo "No sha256sum found in workflows"

echo ""
echo "=== Checking shasum usage in workflows ==="
rg -n 'shasum' .github/workflows/ 2>/dev/null || echo "No shasum found in workflows"

echo ""
echo "=== Checking which runners use sha256sum ==="
# Look for runs-on context near sha256sum usage
rg -B 30 'sha256sum' .github/workflows/mobile-e2e.yml | grep -E '(runs-on|sha256sum)'

Repository: clerk/javascript

Length of output: 639


🏁 Script executed:

# Check the job/runner context for line 245 (first usage)
echo "=== Context around line 245 (first sha256sum usage) ==="
sed -n '200,250p' .github/workflows/mobile-e2e.yml | head -60

echo ""
echo "=== Context around line 644 (second sha256sum usage) ==="
sed -n '610,650p' .github/workflows/mobile-e2e.yml

Repository: clerk/javascript

Length of output: 5219


🏁 Script executed:

# Find all job names and their runners in the workflow
echo "=== All jobs and their runs-on configuration ==="
rg -n 'runs-on:' .github/workflows/mobile-e2e.yml -A 1 -B 5 | grep -E '(^[0-9]+:\s*[a-z\-]+:|runs-on)' | head -40

echo ""
echo "=== Job containing line 245 ==="
sed -n '1,245p' .github/workflows/mobile-e2e.yml | grep -E '^\s{2}[a-z\-]+:\s*$|runs-on:' | tail -5

echo ""
echo "=== Job containing line 644 ==="
sed -n '1,644p' .github/workflows/mobile-e2e.yml | grep -E '^\s{2}[a-z\-]+:\s*$|runs-on:' | tail -5

Repository: clerk/javascript

Length of output: 387


🏁 Script executed:

# Check if there are other macOS jobs in workflows that might provide clues about available tools
echo "=== Other macOS jobs in workflows ==="
rg -n 'macos-' .github/workflows/ | grep runs-on

echo ""
echo "=== Check if there are any shell scripts or setup that installs sha256sum ==="
rg -n 'sha256sum|coreutils|homebrew' .github/workflows/

Repository: clerk/javascript

Length of output: 480


🌐 Web query:

GitHub Actions macOS runner sha256sum availability

💡 Result:

The command sha256sum is not preinstalled on GitHub Actions macOS runners [1][2]. Because macOS is a Unix-based system, it uses the native shasum utility instead [1][2]. To calculate a SHA-256 checksum on macOS runners, you should use the following command [1][2]: shasum -a 256 If your workflow specifically requires sha256sum (e.g., for cross-platform compatibility with scripts written for Linux), you can install it by installing the GNU coreutils package using Homebrew [1][2]: - name: Install coreutils run: brew install coreutils Alternatively, consider updating your scripts to use the native shasum command to avoid the additional installation time [1][2].

Citations:


sha256sum is unavailable on macOS runners and will cause the step to fail.

The ios job runs on macOS-15, which does not have sha256sum installed (it provides shasum instead). The hash computation at line 644 will fail with "command not found".

Use the portable equivalent that works on both macOS and Linux:

Fix
       - name: Compute binary source hash
         id: bin-hash
         run: |
           expo_tree=$(git ls-tree -r HEAD packages/expo .github/workflows/mobile-e2e.yml | grep -v "/dist/")
           qs_tree=$(git -C clerk-expo-quickstart ls-tree -r HEAD NativeComponentQuickstart | grep -vE "node_modules|/android/|/ios/")
-          hash=$(printf '%s\n%s\n' "$expo_tree" "$qs_tree" | sha256sum | cut -c1-16)
+          hash=$(printf '%s\n%s\n' "$expo_tree" "$qs_tree" | shasum -a 256 | cut -c1-16)
           echo "hash=$hash" >> "$GITHUB_OUTPUT"
           echo "Binary source hash: $hash"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Compute binary source hash
id: bin-hash
run: |
expo_tree=$(git ls-tree -r HEAD packages/expo .github/workflows/mobile-e2e.yml | grep -v "/dist/")
qs_tree=$(git -C clerk-expo-quickstart ls-tree -r HEAD NativeComponentQuickstart | grep -vE "node_modules|/android/|/ios/")
hash=$(printf '%s\n%s\n' "$expo_tree" "$qs_tree" | sha256sum | cut -c1-16)
echo "hash=$hash" >> "$GITHUB_OUTPUT"
echo "Binary source hash: $hash"
- name: Compute binary source hash
id: bin-hash
run: |
expo_tree=$(git ls-tree -r HEAD packages/expo .github/workflows/mobile-e2e.yml | grep -v "/dist/")
qs_tree=$(git -C clerk-expo-quickstart ls-tree -r HEAD NativeComponentQuickstart | grep -vE "node_modules|/android/|/ios/")
hash=$(printf '%s\n%s\n' "$expo_tree" "$qs_tree" | shasum -a 256 | cut -c1-16)
echo "hash=$hash" >> "$GITHUB_OUTPUT"
echo "Binary source hash: $hash"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/mobile-e2e.yml around lines 639 - 646, The step "Compute
binary source hash" (id: bin-hash) uses sha256sum which is not available on
macOS runners; replace the sha256sum invocation with the portable shasum -a 256
and extract the first 16 hex chars (e.g. pipe to awk '{print substr($1,1,16)}'
or use cut -c1-16) so the computed value assigned to the hash variable remains
the same and works on both Linux and macOS.

Copy link
Copy Markdown

@swolfand swolfand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Android looks good!

Bandaid pass to stop chasing single-flow timing flakes:
- assert-signed-in: convert all assertVisible to extendedWaitUntil
  (20s for "Manage Profile", 5s for "Sign Out" and "Welcome")
- assert-signed-out: extendedWaitUntil 20s for "Welcome!" instead of
  bare assertVisible
- sign-out-via-button: extendedWaitUntil 20s for "Welcome!" + bump
  email-field wait 10s -> 25s (this is the consistently-slowest path)
- sign-in-email-password: extendedWaitUntil 20s for "Welcome!" + bump
  email-field wait 10s -> 25s, take a debug screenshot before the wait
  so future failures here are diagnosable

assertVisible has no retry; extendedWaitUntil retries until timeout,
which is what we want everywhere in CI where renders are slow.
iOS's system 'Save Password' / iCloud Keychain prompt overlays the home
screen after sign-in. The flow already dismisses Android's Google
Password Manager equivalent; mirror that for the iOS variants
("Save Password", "Strong Password", "AutoFill Passwords") by tapping
the dismissal button ("Not Now", "Never for This Website", "Don't Save").

Also adds two screenshots so the next failure isn't blind:
- debug-04-after-password-continue at the end of sign-in
- debug-assert-signed-in-state at the start of assert-signed-in
Move 'Sign Out' to the lead assertion at 20s timeout. It's the same
shape (TouchableOpacity > Text) that Android already passes against,
so if it fails we know we're not actually signed in; if Sign Out passes
but Manage Profile still times out, we have a specific iOS accessibility
quirk to chase rather than a generic timing issue.

Diagnostic-only reorder — same three assertions, different order.
P1 (cache honesty): the compat-gate pin steps mutate the working tree
(packages/expo/app.plugin.js, packages/expo/android/build.gradle), but the
bin-hash that keys the .app / .apk cache was computed from `git ls-tree -r
HEAD`, which reads committed blobs. A stale cache hit could install an OLD
SDK ref while claiming to test a new one. Fold clerk_ios_ref,
clerk_android_ref, and clerk_android_snapshot_suffix into the hash on both
jobs so the cache key reflects what's actually being built.

P2 (Swift tests on production code): pulled the two pure predicates the
existing tests were mirroring — `isSuccessfulAuth` (from
ClerkAuthWrapperViewController.viewDidDisappear) and the presentWhenReady
guard (from ClerkAuthNativeView) — into ClerkAuthLogic.swift in the pod
source files. ClerkViewFactory.swift now imports ClerkExpo and calls
ClerkAuthSessionLogic; ClerkExpoModule.swift's view layer now calls
ClerkPresentationLogic. The XCTest files `@testable import ClerkExpo`
and exercise the same public symbols production runs against — no more
duplicated logic to drift. The maxPresentationAttempts constant lives on
the production type so a bump can't silently break the test.

P2 (forgot-password OAuth automated): split google-sso-from-forgot-password
into two files. The original bug was that tapping "Sign in with Google"
from the forgot-password screen was a silent no-op on iOS — that part is
now automated by asserting the system OAuth presentation appears
("Wants to Use" / "accounts.google.com" / "Continue" / "google.com").
The actual Google credentialed completion still needs real OAuth and lives
in google-sso-from-forgot-password-manual.yaml with the `manual` tag, so
the default workflow exclude (manual,skip) keeps it out of CI but it's
runnable as a one-off.

P2 (local scripts pre-filter): Maestro's `--exclude-tags` is a no-op when
explicit file paths are passed, which run-ios.sh / run-android.sh /
run-regressions.sh all do. Added scripts/lib/filter-flows.sh — a shared
helper that scans each flow's YAML frontmatter for tag-list entries
matching the excluded set — and routed all three scripts through it
before they invoke maestro.
The Android pre-filter was only excluding "flakyAndroid" on top of the
user-supplied EXCLUDE_TAGS — it never excluded the platform-scope tag
"iosOnly", so any flow tagged iosOnly was being run on the Android
emulator anyway. Caught by the new google-sso-from-forgot-password
flow, which is correctly tagged iosOnly but failed on Android because
the system OAuth presentation strings ("Wants to Use" /
"accounts.google.com" / "Continue") never appear there.

iOS already had the symmetric guard ("$EXCLUDE_TAGS,androidOnly").
Android side: artifact screenshots from yesterday's gate-run failure
showed the AuthView fully rendered with the email field's placeholder
text on screen, but Maestro's accessibility-tree query for it timed out
after 25s. That's a known cold-emulator quirk — the screen renders
before the a11y tree is fully populated, and whichever flow happens to
run first eats the cost. Added a one-shot warmup against
flows/common/_warmup.yaml before the per-flow xargs loop so the JS
bundle and a11y tree are primed when the real flows start. Also added
`| sort` to the find pipeline so flow ordering is deterministic across
runs.

iOS side: `expo run:ios` BUILT and INSTALLED successfully, then tried
to deep-link com.<bundle>://expo-development-client/?url=http://<LAN-IP>:8081
to launch the dev launcher. On GitHub-hosted macos-15 runners the LAN
IP is unreachable from the simulator and `xcrun simctl openurl` times
out at 60s, exiting the expo CLI with code 1 even though the .app is
sitting in DerivedData ready to use. We don't need the post-install
launch (Maestro re-installs and opens the app cleanly later), so trap
the exit code and let the .app-exists check below decide whether to
proceed. Caught by today's manual full e2e run where the iOS job failed
with "Operation timed out" right after "Build Succeeded".
Cancelled run 26253322728 showed every flow passing in order, but the
job hit its 60-minute wall at the 22-minute install/build mark plus
~5-8 minutes per top-level flow on iOS sim. iOS is inherently slower
than Android (ASWebAuthenticationSession setup, simctl install per
launch, SwiftUI a11y-tree population) — the suite just runs out the
clock on cache-miss days.

Bumping the wall to 90 to clear that comfortably. Drop back down once
we shard or persist DerivedData across runs.
Android compat-gate dispatched run 26258070936 failed in 11 seconds with:
  java.io.IOException: Downloading from
  https://services.gradle.org/distributions/gradle-9.5.0-bin.zip
  failed: timeout (10000ms)

That's the default 10-second Gradle wrapper download socket timeout
biting on a slow services.gradle.org response. The publish step never
ran a single task. Two-pronged fix:

1. Bump Gradle's HTTP timeouts to 60s via GRADLE_OPTS so transient
   slow-fetch periods don't trip a single-request hard fail.
2. Wrap the gradlew invocation in a 3-attempt retry loop with a 10s
   pause between attempts to absorb single-shot upstream blips.

The retry is scoped to the snapshot-publish step only; build/test
steps below have their own caching/retry semantics.
Bare assertVisible at the end of the flow has no retry, so a slow
emulator that hasn't finished AuthView render by t=10s post-launch
fails the assertion even though the screen shows up a few seconds
later. The no-flash regression check is the cold-launch-immediate
screenshot captured before this assertion; the assertion only confirms
we landed on the AuthView at all. Switch to extendedWaitUntil 30s to
match open-app.yaml's pattern. Same flow has been intermittently green
for weeks because the timing race resolves differently per boot.
Android gate run 26303048981 failed on the first flow's launchApp with:
  Launch app "com.clerk.clerkexpoquickstart" with clear state... FAILED

The 8s-later second flow's launchApp clearState succeeded. Root cause:
after the warmup completes, the app is still foregrounded; clearState
under the hood is `pm clear` (Android) / simctl clear (iOS), which
silently fails when the package is in use. Subsequent flows work
because Maestro's session teardown stops the app between invocations.

Bridge that gap explicitly — force-stop the app between warmup and
the per-flow loop on both platforms (adb am force-stop on Android,
xcrun simctl terminate on iOS). Both are no-ops if the app is already
stopped.
Stability re-run 2 hit a different Maestro flake on the 5th-or-so
Android flow: "Enter your email or username" extendedWaitUntil timed
out at 25s after launchApp clearState even though the AuthView's
Welcome header was visible (Android Compose accessibility tree lags
visual render on some boots). We've now fixed three separate timing
flakes; each one was a real defect that needed a real fix, but the
underlying pattern is that any individual maestro test invocation has
a non-trivial chance of losing a single-shot timing race on the CI
emulator.

Maestro doesn't expose a --retries flag in the version we ship. Wrap
the per-flow xargs invocation in a 2-attempt retry loop via
`xargs -I FLOW bash -c '...'`: first failure forces-stop the app,
sleeps 10s, retries; second consecutive failure is a real failure
and propagates the non-zero. Catches single-shot timing flakes
without masking genuine regressions (a real bug fails both attempts).
Applied symmetrically on iOS and Android.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants